Repository navigation
gh-162016: Fix communicate() hang when a full stdin pipe is reported writable - #162058
JohnnyWilson16 wants to merge 6 commits into
Conversation
On some macOS versions, poll() reports a full pipe as writable. The POSIX communicate() loop then called a blocking os.write() on the child's stdin, which hung past the timeout and could deadlock against a child that was blocked writing to stdout. Set stdin to non-blocking around _communicate_io_posix(), treat BlockingIOError as "not ready yet", and restore blocking mode if stdin is still open, so a resumed communicate() and direct proc.stdin writes work as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gpshead
left a comment
There was a problem hiding this comment.
this attempt at a solution winds up busy-looping wasting CPU under the macOS wide global max pipe buffers limit (see analysis linked from the issue) that triggers the OS bug. if we go this route, we'd need a backoff mechanism as part of the BlockingIOError (EAGAIN / EWOULDBLOCK) exception handler.
This comment was marked as resolved.
This comment was marked as resolved.
With stdin non-blocking, a pipe that poll() keeps reporting as writable while it is full made communicate() spin: select() returned at once, write() raised BlockingIOError, and the loop repeated, using a full core until the child read its input or the timeout expired. After a write that would block, take stdin out of the selector for a delay that doubles from 1 ms to 50 ms and resets on the next successful write. The other pipes are still serviced during the delay, and any activity on them ends it early, since output from the child suggests it has also consumed input. Measured on macOS 27.2 with the system-wide pipe buffer limit reached, for a child that never reads stdin and communicate(timeout=3): 3.00 s of CPU before this change, 0.01 s after. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Use "is not None" when restoring stdin's blocking mode, matching the check made before the try block. Reword the NEWS entry to say that this works around a macOS and FreeBSD kernel bug that appears under system-wide pipe buffer memory pressure, most likely on loaded hosts with many pipes in use. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Lead with the affected API and what a user would notice, and drop the description of how the workaround is implemented. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Explain what stdin_paused_for holds, say why stdin's blocking mode is restored in terms of what later callers expect, and describe the test's blocking case without referring to "the fix". Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The test selector waited 10 ms in every select() call, which hid a busy loop. It now reports stdin as writable without waiting and counts how often it did so for a pipe that was not, so the timeout test can assert that communicate() does not keep retrying a full pipe. Add a test for the case where stdin is full and is the only pipe left: the child closes stdout before it reads its input, and checks that it received all of it. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
Thanks for the analysis @gpshead and for pushing the backoff. You were right about the spin. I pulled your commits and checked them on macOS 26.6 arm64 (pydebug build). I reproduced the real kernel condition by filling 257 pipes until new pipes stayed at 512 bytes, then ran the issue's Same pressure, child that never reads stdin,
The backoff looks right to me. stdout and stderr stay registered during the pause, so the child can still drain its output, and the |
On macOS and FreeBSD,
poll()andselect()can report a pipe as writable when a write to it would block.communicate()assumed a writable event meant aPIPE_BUF-sized write could not block, so its blockingos.write()to the child's stdin could stall pasttimeout, or deadlock against a child that was itself blocked writing to stdout.On macOS this happens once system-wide pipe buffer memory reaches its fixed 16 MiB limit. A new pipe then stays at the size of its first write (512 bytes for
communicate()), while readiness is computed as if it could still grow to 16 KiB. It is most likely to be seen on loaded hosts with many pipes in use. The FreeBSD case is a narrower window found by reading its kernel source and has not been reproduced.Changes
Popen._communicate()makes stdin non-blocking while_communicate_io_posix()runs and restores blocking mode afterwards if stdin is still open. A resumedcommunicate()afterTimeoutExpiredand directproc.stdinwrites behave as before._communicate_io_posix()handlesBlockingIOErrorby taking stdin out of the selector for a delay that doubles from 1 ms to 50 ms and resets on the next successful write. A timed retry is needed because the kernel keeps reporting the full pipe as writable, so there is no event to wait for. The other pipes are still serviced during the delay, and activity on them ends it early.Nothing changes when the kernel reports readiness truthfully. In the affected state, a child that reads stdin but writes no output receives its input more slowly than with a blocking write: about 3 MiB/s in the measurement below.
Tests
Both tests replace
_PopenSelectorwith one that always reports stdin as writable, so they do not need to put the host into the affected state.test_communicate_timeout_stdin_spuriously_writable: the child never reads stdin.communicate()must raiseTimeoutExpiredon time, must not retry the full pipe in a busy loop, and must leave stdin blocking.test_communicate_stdin_spuriously_writable_last_pipe: the child closes stdout before reading stdin, so stdin is full and is the only pipe left. All input must still arrive.Verified against the real kernel behavior
On macOS 27.2 (arm64), with the limit reached by one process holding 255 pipes filled to 64 KiB:
/bin/catround trip,timeout=2)communicate(timeout=3)TimeoutExpiredafter 6.0 s, when the child exitsTimeoutExpiredafter 3.0 s, 0.01 s CPU